FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence

One of the fundamental issues for Federated Learning (FL) is data heterogeneity, which causes accuracy degradation, slow convergence, and communication bottleneck issue. Although impact heterogeneity on supervised FL has been widely studied, related investigation Reinforcement (FRL) still in its infancy. In this paper, we first define type level FRL systems. By inspecting connection between global local objective functions, prove that training can benefit objective, if update properly penalized by total variation (TV) distance policies. A necessary condition policy to be learn-able from environments also derived, directly level. Based theoretical result, a Kullback-Leibler (KL) divergence based penalty proposed constrain model outputs distribution space convergence proof algorithm provided. jointly penalizing with each iteration penalty, method achieves better trade-off speed (step size) convergence. Experiment results two popular (RL) experiment platforms demonstrate advantage over existing methods accelerating stabilizing process heterogeneous data.